Head mounted display providing eye gaze calibration and control method thereof

ABSTRACT

A control method of a head mounted display (HMD) is disclosed. The control method of the HMD includes detecting a first route of a first moving object and a second route of a second moving object in front of the HMD, detecting a third route along which user&#39;s eye gaze moves, setting the first moving object as a reference object if the detected first route is substantially identical with the detected third route, setting the second moving object as the reference object if the detected second route is substantially identical with the detected third route, and performing an eye gaze calibration based on a route of the set reference object and the detected third route.

Pursuant to 35 U.S.C. §119(a), this application claims the benefit ofpriority to korea Applications No. 10-2013-0153840, filed on Dec. 11,2013, and No. 10-2013-0002558, filed on Jan. 9, 2013, the contents ofwhich are incorporated by reference herein in their entirety.

This application is a Continuation-in-Part of copending U.S. applicationSer. No. 13/772,444, filed on Feb. 21, 2013, the contents of which areincorporated by reference herein in their entirety.

Pursuant to 35 U.S.C. §119(a), this application claims the benefit ofPCT Application No. PCT/KR2013/001655, filed on Feb. 28, 2013, thecontents of which are incorporated by reference herein in theirentirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present specification relates to a head mounted display (HMD), andmore particularly, to an HMD that is capable of calibrating a user's eyegaze direction and a control method thereof.

2. Discussion of the Related Art

A head mounted display (HMD) may be worn by a user like glasses. Theuser may receive digital information displayed on the HMD. The HMD maycalibrate a user's eye gaze direction to detect which the user gazes at.The HMD may adjust the position of the digital information displayed onthe HMD based on the user's eye gaze direction and provide the user withdigital information including an adjusted display position. In order toperform eye gaze calibration, it is necessary for the HMD to provide aneye gaze calibration screen to the user such that the user views the eyegaze calibration screen. Also, it is necessary for the user to perform aprocess for an eye gaze calibration before using the HMD. As a result,it is necessary for the user to have time to perform the eye gazecalibration of the HMD, which is troublesome.

SUMMARY OF THE INVENTION

Accordingly, the present specification is directed to a head mounteddisplay (HMD) providing an eye gaze calibration and a control methodthereof that substantially obviate one or more problems due tolimitations and disadvantages of the related art.

An object of the present specification is to provide an eye gazecalibration method of an HMD that does not require time to calibrate auser's eye gaze direction.

Additional advantages, objects, and features of the specification willbe set forth in part in the description which follows and in part willbecome apparent to those having ordinary skill in the art uponexamination of the following or may be learned from practice of thespecification. The objectives and other advantages of the specificationmay be realized and attained by the structure particularly pointed outin the written description and claims hereof as well as the appendeddrawings.

To achieve these objects and other advantages and in accordance with thepurpose of the invention, as embodied and broadly described herein, anHMD includes a camera unit configured to detect a first route of a firstmoving object and a second route of a second moving object in front ofthe HMD, an eye gaze detection unit configured to detect a third routealong which user's eye gaze moves, and a controller configured toperform eye gaze calibration based on a route of a reference object andthe detected third route, wherein the controller sets the first movingobject as the reference object if the detected first route issubstantially identical with the detected third route and the controllersets the second moving object as the reference object if the detectedsecond route is substantially identical with the detected third route.

In another aspect of the present invention, a control method of a HMDincludes detecting a first route of a first moving object and a secondroute of a second moving object in front of the HMD, detecting a thirdroute along which user's eye gaze moves, setting the first moving objectas a reference object if the detected first route is substantiallyidentical with the detected third route, setting the second movingobject as the reference object if the detected second route issubstantially identical with the detected third route, and performingeye gaze calibration based on a route of the set reference object andthe detected third route.

It is to be understood that both the foregoing general description andthe following detailed description of the present specification areexemplary and explanatory and are intended to provide furtherexplanation of the specification as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the specification and are incorporated in andconstitute a part of this application, illustrate embodiment(s) of thespecification and together with the description serve to explain theprinciple of the specification. In the drawings:

FIG. 1 is a view showing an eye gaze calibration method of a headmounted display (HMD) according to an embodiment of the presentspecification;

FIG. 2 is a view showing an eye gaze calibration method of an HMD usinga reference point according to an embodiment of the presentspecification;

FIG. 3 is a view showing examples of a reference object and a referencepoint according to an embodiment of the present specification;

FIG. 4 is a view showing examples of a reference object and a referencepoint according to another embodiment of the present specification;

FIG. 5 is a view showing a method of displaying an augmented realitymessage using eye gaze calibration according to an embodiment of thepresent specification;

FIG. 6 is a view showing a method of correcting an intersection point ofthe HMD according to an embodiment of the present specification;

FIG. 7 is a view showing an eye gaze calibration method of an HMD usinga moving object according to an embodiment of the present specification;

FIG. 8 is a view showing an eye gaze calibration method of an HMD usinga moving object according to another embodiment of the presentspecification;

FIG. 9 is a view showing a method of performing eye gaze calibrationwith respect to both eyes of a user using a reference point;

FIG. 10 is a block diagram showing an HMD according to an embodiment ofthe present specification;

FIG. 11 is a flowchart showing an eye gaze calibration method accordingto an embodiment of the present specification;

FIG. 12 is a flowchart showing a method of deciding a reference objectaccording to an embodiment of the present specification;

FIG. 13 is a flowchart showing a method of deciding a reference objectaccording to another embodiment of the present specification; and

FIG. 14 is a flowchart showing a method of deciding a reference objectaccording to a further embodiment of the present specification.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the preferred embodiments of thepresent specification, examples of which are illustrated in theaccompanying drawings. It should be noted herein that these embodimentsare only for illustrative purposes and the protection scope of thespecification is not limited or restricted thereto.

Terms used in this specification are general terms selected inconsideration of functions and widely used at the present time. However,such terms may vary depending upon intentions of those skilled in theart to which the present specification pertains, usual practices, orappearance of new technology. In a specific case, some terms may beselected by the applicant of the present application. In this case,meanings of such terms will be described in corresponding paragraphs ofthe specification. Therefore, it should be noted that terms used in thisspecification be interpreted based on real meanings of the terms and thedisclosure of the present specification, not simple names of the terms.

In embodiments of the present specification, a head mounted display(HMD) may include a computer that a user wears on the head, i.e. awearable computer. In embodiments of the present specification, avirtual object may be an object displayed on a display unit of the HMD.For example, the HMD may display virtual objects, such as an applicationicon, an alarm message, an augmented reality (AR) message, a navigationdirection indicator, and a notification message, on the display unit.

Also, in embodiments of the present specification, a real object may bean object contained in the surroundings sensed by a camera unit of theHMD. For example, the real object may include various objects containedin a visual field of a user in a state in which the use wears the HMD.The HMD may sense a region corresponding to the visual field of the useras an image using the camera unit. Also, the HMD may distinguish objectscontained in the sensed image using an image processing unit.

Also, in embodiments of the present specification, a reference objectmay be an object at which a user gazes when the HMD calibrates a user'seye gaze direction. The reference object may be a criterion for eye gazecalibration. Also, a reference point may be a portion of the referenceobject. The reference point may be used to improve accuracy of eye gazecalibration.

FIG. 1 is a view showing an eye gaze calibration method of an HMDaccording to an embodiment of the present specification. The HMD maydetect a reference object and a user's eye gaze direction as a referencepoint for eye gaze calibration to perform the eye gaze calibration.According to embodiments, the HMD may decide a reference point as thereference point instead of the reference object. The HMD may detect atleast one real object or a portion of the body of a user contained inthe surroundings using an image processing unit. The HMD may detect anevent on the real object. The HMD may detect a user's operation ofcontrolling or instructing a real object as an event on the real object.The HMD may decide the real object, on which the event has beendetected, as a reference object of the eye gaze calibration. The HMD mayperform eye gaze calibration based on the position of the referenceobject and a user's eye gaze direction in which the user gazes at thereference object.

The HMD 10 may detect at least one real object 11-1 and 11-2 containedin the surroundings. Also, the HMD 10 may detect a portion of the bodyof the user. In FIG. 1, a hand 12 of the user is shown as an example ofa portion of the body of the user. The HMD 10 may distinguish between ahand of the user and a hand of another person using at least one of ashape of the hand of the user and a gesture of the hand of the user. TheHMD 10 may detect an event on the real object by the hand of the user.The HMD 10 may detect a user's operation of touching the real objectusing the hand or instructing the real object as the event on the realobject. The HMD 10 may decide the real object 11-1, on which the eventhas been detected, as a reference object of the eye gaze calibration.Also, the HMD 10 may decide a portion of the reference object, on whichthe event has been detected, as a reference point. The HMD 10 may detecta user's eye gaze direction. The HMD 10 may detect the position of apupil 13 of the user to decide the user's eye gaze direction.

The HMD 10 may perform eye gaze calibration based on the position of thedecided reference object and the position of the pupil 13 of the user.Alternatively, the HMD 10 may perform eye gaze calibration based on theposition of the decided reference point and the position of the pupil 13of the user. If the HMD 10 performs the eye gaze calibration based onthe position of the reference point, the eye gaze calibration may beperformed with higher accuracy than in a case in which the HMD 10performs the eye gaze calibration based on the position of the referenceobject.

The HMD 10 may set a virtual line linking the pupil 13 of the user tothe reference object and calculate an intersection point 14 between thevirtual line and the HMD. Alternatively, the HMD 10 may set a virtualline linking the pupil 13 of the user to the reference point andcalculate the intersection point 14 between the virtual line and theHMD. The HMD 10 may decide the position of digital information to bedisplayed on the HMD based on the calculated position of theintersection point 14.

As described above, the HMD 10 has an effect in that the HMD 10 mayperform eye gaze calibration while a user wears and uses the HMD 10without using an additional calibration screen or calibration time.

FIGS. 2(a) and 2(b) are views showing an eye gaze calibration method ofan HMD using a reference point according to an embodiment of the presentspecification. The HMD 10 may decide a portion of a reference object asa reference point such that the reference point is used to perform theeye gaze calibration. The HMD 10 may detect an event on the referenceobject by a hand of the user. The HMD 10 may decide a point of thereference object on which the event has been generated as the referencepoint.

Referring to FIG. 2(a), the HMD 10 may decide a reference object 11 asdescribed with reference to FIG. 1. The HMD 10 may detect a hand 12 ofthe user and detect a first event for the hand 12 of the user to touchthe right side upper end of the reference object 11. The HMD 10 maydecide the right side upper end of the reference object, on which thefirst event has been generated, as a first reference point 23-1 using acamera unit 21 and an image processing unit. The HMD 10 may decide auser's eye gaze direction in which the user gazes at the first referencepoint 23-1 using an eye gaze detection unit 22. The HMD 10 may detectthe position of a pupil 13 of the user to decide the user's eye gazedirection. The HMD 10 may set a virtual line 25 linking the pupil 13 ofthe user to the first reference point 23-1 and calculate the position ofa first intersection point 24-1 between the virtual line 25 and the HMD.That is, the HMD 10 may perform eye gaze calibration based on theposition of the pupil 13 of the user and the position of the firstreference point 23-1 and calculate the position of the firstintersection point 24-1 as mentioned above as the result of the eye gazecalibration. The HMD 10 may decide the position of a virtual object tobe displayed on the HMD based on the calculated position of the firstintersection point 24-1.

In a case in which the detected position of the hand of the user ischanged, the HMD 10 may detect this as another event, which is differentfrom the existing event. Referring to FIG. 2(b), the HMD 10 may detect asecond event for the hand 12 of the user to touch the middle lower endof the reference object 11. The HMD 10 may decide the middle lower endof the reference object 11, on which the second event has beengenerated, as a second reference point 23-2 using the camera unit 21 andthe image processing unit. The HMD 10 may decide a user's eye gazedirection in which the user gazes at the second reference point 23-2using the eye gaze detection unit 22. The HMD 10 may detect the positionof the pupil 13 of the user to decide the user's eye gaze direction. TheHMD 10 may set a virtual line 25 linking the pupil 13 of the user to thesecond reference point 23-2 and calculate the position of a secondintersection point 24-2 between the virtual line 25 and the HMD. Thatis, the HMD 10 may perform eye gaze calibration based on the position ofthe pupil 13 of the user and the position of the second reference point23-2 and calculate the position of the second intersection point 24-2 asmentioned above as the result of the eye gaze calibration. The HMD 10may decide the position of a virtual object to be displayed on the HMDbased on the calculated position of the second intersection point 24-2.

FIGS. 3(a) and 3(b) are views showing examples of a reference object anda reference point according to an embodiment of the presentspecification. The HMD 10 may detect a real object and a hand of theuser. The HMD 10 may detect an event on the real object by the hand ofthe user. The HMD 10 may decide the real object, on which the event hasbeen detected, as a reference object. The HMD 10 may decide a point ofthe reference object, on which the event has been detected, as areference point. The HMD 10 may use at least one of the decidedreference object and the decided reference point to perform the eye gazecalibration.

FIG. 3(a) shows eye gaze calibration using a writing operation performedby the user using a pen. The HMD 10 may detect a pen 31 and a notebook32 as a real object using the camera unit 21 and the image processingunit. Also, the HMD 10 may detect a hand 12 of the user performing awriting operation while holding the pen 31. The HMD 10 may detect anoperation of performing the writing operation using the hand 12 of theuser while holding the pen 31 as an event. As a result, the HMD 10 maydecide the pen 31 as a reference object. Also, the HMD 10 may decide thetip or a pen point of the pen 31 contacting the notebook 32 as areference point.

In a case in which the eye gaze calibration is performed using thereference object, the HMD 10 may perform eye gaze calibration using theposition of the pen 31, which is the reference object, and a user's eyegaze direction. The HMD 10 may detect the position of the pupil 13 ofthe user using the eye gaze detection unit 22 and decide the user's eyegaze direction based thereupon. The HMD 10 may set a virtual line 35linking the pupil 13 of the user to the reference object 31 andcalculate the position of an intersection point 34 between the virtualline 35 and the HMD. That is, the HMD 10 may perform eye gazecalibration based on the position of the pupil 13 of the user and theposition of the reference object 31 and calculate the position of theintersection point 34 as mentioned above as the result of the eye gazecalibration. The HMD 10 may decide the position of a virtual object tobe displayed on the HMD based on the calculated position of theintersection point 34.

In a case in which the eye gaze calibration is performed using thereference point, the HMD 10 may decide the tip or the pen point of thepen 31, which is the reference object, as a reference point 33. The HMD10 may perform eye gaze calibration using the position of the referencepoint and a user's eye gaze direction. The HMD 10 may detect theposition of the pupil 13 of the user using the eye gaze detection unit22 and decide the user's eye gaze direction based thereupon. The HMD 10may set a virtual line 35 linking the pupil 13 of the user to thereference point 33 and calculate the position of an intersection point34 between the virtual line 35 and the HMD. That is, the HMD 10 mayperform eye gaze calibration based on the position of the pupil 13 ofthe user and the position of the reference point 33 and calculate theposition of the intersection point 34 as mentioned above as the resultof the eye gaze calibration. The HMD 10 may decide the position of avirtual object to be displayed on the HMD based on the calculatedposition of the intersection point 34.

FIG. 3(b) shows eye gaze calibration using a user's operation ofcontrolling a portable device. The HMD 10 may detect a portable device36 as a real object using the camera unit 21 and the image processingunit. Also, the HMD 10 may detect a hand 12-2 of the user gripping theportable device 36 and a hand 12-1 of the user touching the portabledevice 36. The HMD 10 may detect an operation of touching the portabledevice 36 using at least one of the hand 12-1 of the user and anoperation of gripping the portable device 36 using the hand 12-2 of theuser as an event. As a result, the HMD 10 may decide the portable device36 as a reference object. Also, the HMD 10 may decide a portion of theportable device 36, which the hand 12-1 of the user has touched, as areference point.

In a case in which the eye gaze calibration is performed using thereference object, the HMD 10 may perform eye gaze calibration using theposition of the portable device 36, which is the reference object, and auser's eye gaze direction. The HMD 10 may detect the position of thepupil 13 of the user using the eye gaze detection unit 22 and decide theuser's eye gaze direction base thereupon. The HMD 10 may set a virtualline 39 linking the pupil 13 of the user to the reference object 36 andcalculate the position of an intersection point 38 between the virtualline 39 and the HMD. That is, the HMD 10 may perform eye gazecalibration based on the position of the pupil 13 of the user and theposition of the reference object 36 and calculate the position of theintersection point 38 as mentioned above as the result of the eye gazecalibration. The HMD 10 may decide the position of a virtual object tobe displayed on the HMD based on the calculated position of theintersection point 38.

In a case in which the eye gaze calibration is performed using thereference point, the HMD 10 may decide a point of the reference object,i.e. the portable device 36, on which the event has been generated, as areference point 37. For example, in a case in which the hand of the usertouches an object displayed on the portable device 36, a point of theobject, which the user has touched, may be decided as a reference point37. The HMD 10 may perform eye gaze calibration using the position ofthe reference point and a user's eye gaze direction. The HMD 10 maydetect the position of the pupil 13 of the user using the eye gazedetection unit 22 and decide the user's eye gaze direction basedthereupon. The HMD 10 may set a virtual line 39 linking the pupil 13 ofthe user to the reference point 37 and calculate the position of anintersection point 38 between the virtual line 39 and the HMD. That is,the HMD 10 may perform eye gaze calibration based on the position of thepupil 13 of the user and the position of the reference point 37 andcalculate the position of the intersection point 38 as mentioned aboveas the result of the eye gaze calibration. The HMD 10 may decide theposition of a virtual object to be displayed on the HMD based on thecalculated position of the intersection point 38.

FIGS. 4(a) and 4(b) are views showing examples of a reference object anda reference point according to another embodiment of the presentspecification. The HMD 10 may detect a real object and a hand of theuser. The HMD 10 may detect an event on the real object by the hand ofthe user. The HMD 10 may decide the real object, on which the event hasbeen detected, as a reference object. The HMD 10 may decide a point ofthe reference object, on which the event has been detected, as areference point. The HMD 10 may use at least one of the decidedreference object and the decided reference point to perform the eye gazecalibration. Also, the HMD 10 may detect the hand of the user as a realobject. The HMD 10 may detect at least one of a gesture of the hand ofthe user and a shape of the hand of the user and decide the detectedhand of the user as a reference object together with a predeterminedgesture or shape of the hand of the user. Also, the HMD 10 may decide aportion of the hand of the user as a reference point.

FIG. 4(a) shows eye gaze calibration using a user's operation ofcontrolling a switch. The HMD 10 may detect a switch 41 as a real objectusing the camera unit 21 and the image processing unit. Also, the HMD 10may detect a hand 12 of the user to control the switch 41. The HMD 10may detect an operation of controlling the switch 41 using the hand 12of the user as an event. As a result, the HMD 10 may decide the switch41 as a reference object. Also, the HMD 10 may decide a portion of theswitch, which the hand 12 of the user has touched, as a reference point.

In a case in which the eye gaze calibration is performed using thereference object, the HMD 10 may perform eye gaze calibration using theposition of the switch 41, which is the reference object, and a user'seye gaze direction. The HMD 10 may detect the position of the pupil 13of the user using the eye gaze detection unit 22 and decide the user'seye gaze direction based thereupon. The HMD 10 may set a virtual line 43linking the pupil 13 of the user to the reference object 41 andcalculate the position of an intersection point 44 between the virtualline 43 and the HMD. That is, the HMD 10 may perform eye gazecalibration based on the position of the pupil 13 of the user and theposition of the reference object 41 and calculate the position of theintersection point 44 as mentioned above as the result of the eye gazecalibration. The HMD 10 may decide the position of a virtual object tobe displayed on the HMD based on the calculated position of theintersection point 44.

In a case in which the eye gaze calibration is performed using thereference point, the HMD 10 may decide a portion of the switch 41, whichis the reference object, which the hand of the user has touched, as areference point 42. The HMD 10 may perform eye gaze calibration usingthe position of the reference point and a user's eye gaze direction. TheHMD 10 may detect the position of the pupil 13 of the user using the eyegaze detection unit 22 and decide the user's eye gaze direction basedthereupon. The HMD 10 may set a virtual line 43 linking the pupil 13 ofthe user to the reference point 42 and calculate the position of anintersection point 44 between the virtual line 43 and the HMD. That is,the HMD 10 may perform eye gaze calibration based on the position of thepupil 13 of the user and the position of the reference point 42 andcalculate the position of the intersection point 44 as mentioned aboveas the result of the eye gaze calibration. The HMD 10 may decide theposition of a virtual object to be displayed on the HMD based on thecalculated position of the intersection point 44.

Also, the HMD 10 may detect the hand of the user as a real object. TheHMD 10 may detect at least one of a gesture of the hand of the user anda shape of the hand of the user and decide the detected hand of the useras a reference object together with at least one of a predeterminedgesture and shape of the hand of the user. Also, the HMD 10 may decide aportion of the hand of the user as a reference point.

FIG. 4(b) shows eye gaze calibration using a hand of the user as areference object. The HMD 10 may detect a hand 12 of the user as a realobject using the camera unit 21 and the image processing unit. The HMD10 may detect at least one of a gesture and a shape indicated by thehand 12 of the user as an event. The HMD 10 may store at least one of agesture and a shape of the hand of the user for the eye gazecalibration. The HMD 10 may compare the gesture indicated by the hand 12of the user with the stored gesture to decide whether the event has beengenerated. Upon determining that the gesture indicated by the hand 12 ofthe user matches the stored gesture, the HMD 10 may detect this as eventgeneration. Also, the HMD 10 may compare the shape indicated by the hand12 of the user with the stored shape of the hand to decide whether theevent has been generated. Upon determining that the shape indicated bythe hand 12 of the user matches the stored shape of the hand, the HMD 10may detect this as event generation.

When the HMD 10 detects an event based on the gesture or the shape ofthe hand 12 of the user, the HMD 10 may decide the hand 12 of the useras a reference object. Also, the HMD 10 may decide a portion of the hand12 of the user as a reference point. For example, the HMD 10 may decidethe tip of a finger of the user as a reference point 45. The HMD 10 maydecide which portion of the hand of the user becomes a reference pointwith respect to a plurality of gestures or hand shapes. That is, in acase in which a first gesture has been detected, the HMD 10 may decide afirst portion of the hand of the user as a reference point, and, in acase in which a second gesture has been detected, the HMD 10 may decidea second portion of the hand of the user as a reference point. Also, ina case in which a first hand shape has been detected, the HMD 10 maydecide a first portion of the hand of the user as a reference point,and, in a case in which a second hand shape has been detected, the HMD10 may decide a second portion of the hand of the user as a referencepoint. For example, in a case in which the HMD 10 has detected aV-shaped hand gesture as shown in FIG. 4(b), the HMD 10 may decide thetip of the index finger as a reference point. On the other hand, in acase in which the HMD 10 has detected a hand with only the thumbstretched upward, the HMD 10 may decide a nail of the thumb as areference point.

The HMD 10 may perform eye gaze calibration using the position of thereference point and a user's eye gaze direction. The HMD 10 may detectthe position of the pupil 13 of the user using the eye gaze detectionunit 22 and decide the user's eye gaze direction based thereupon. TheHMD 10 may set a virtual line 46 linking the pupil 13 of the user to thereference point 45 and calculate the position of an intersection point47 between the virtual line 46 and the HMD. That is, the HMD 10 mayperform eye gaze calibration based on the position of the pupil 13 ofthe user and the position of the reference point 45 and calculate theposition of the intersection point 47 as mentioned above as the resultof the eye gaze calibration. The HMD 10 may decide the position of avirtual object to be displayed on the HMD based on the calculatedposition of the intersection point 47.

FIGS. 5(a) and 5(b) are views showing a method of displaying anaugmented reality (AR) message using eye gaze calibration according toan embodiment of the present specification. An HMD may display anaugmented reality message using eye gaze calibration. The HMD mayprovide a left eye and a right eye of a user with a left image and aright image, respectively, to display an image having depth. That is,the HMD may provide the user with a three-dimensional image usingbinocular disparity between the left image and the right image.

When a reference point is decided by a hand of the user, the HMD maymeasure the distance from the HMD to the reference point using adistance measurement unit. The HMD may decide depth of an augmentedreality message based on the measured distance from the HMD to thereference point. As a result, the HMD may display an augmented realitymessage based on the depth of the reference point.

FIG. 5(a) is a view showing a method of performing eye gaze calibrationwith respect to both eyes of a user using a reference point. The HMD mayinclude a left subunit 10-1 to provide a graphical user interface to aleft eye of the user and a right subunit 10-2 to provide a graphicaluser interface to a right eye of the user. Each subunit of the HMD mayinclude a camera unit and an eye gaze detection unit. Hereinafter,particulars common to the left subunit and the right subunit will bedescribed in terms of the HMD.

The HMD 10-1 and 10-2 may detect a reference object 51 and a hand 12 ofthe user. The HMD 10-1 and 10-2 may detect an event in which the hand 12of the user touches a portion of the reference object 51. The HMD 10-1and 10-2 may decide a point of the reference object, on which the eventhas been generated, as a reference point 52 using a camera unit and animage processing unit. The HMD 10-1 and 10-2 may decide a user's eyegaze direction in which the user gazes at the reference point 52 usingan eye gaze detection unit. The HMD may detect the position of a pupil13 of the user to decide the user's eye gaze direction.

The left subunit 10-1 of the HMD may set a first virtual line 54-1linking a pupil 13-1 of the left eye of the user to the reference point52 and calculate the position of a first intersection point 53-1 betweenthe first virtual line 54-1 and the left subunit 10-1. That is, the HMDmay perform eye gaze calibration based on the position of the pupil 13-1of the left eye of the user and the position of the reference point 52and calculate the position of the first intersection point 53-1 asmentioned above as the result of the eye gaze calibration. The HMD maydecide the position of a left image to be displayed on the left subunit10-1 based on the calculated position of the first intersection point53-1.

The right subunit 10-2 of the HMD may set a first virtual line 54-2linking a pupil 13-2 of the right eye of the user to the reference point52 and calculate the position of a second intersection point 53-2between the first virtual line 54-2 and the right subunit 10-2. That is,the HMD may perform eye gaze calibration based on the position of thepupil 13-2 of the right eye of the user and the position of thereference point 52 and calculate the position of the second intersectionpoint 53-2 as mentioned above as the result of the eye gaze calibration.The HMD may decide the position of a right image to be displayed on theright subunit 10-2 based on the calculated position of the secondintersection point 53-2.

The HMD may measure the distance from the HMD to the reference pointusing a distance measurement unit. The distance measurement unit maymeasure the distance from the HMD to the reference point using at leastone selected from among an ultrasonic sensor, a laser sensor, and aninfrared sensor. The HMD may decide depth of an augmented realitymessage based on the measured distance from the MID to the referencepoint. For example, the HMD may set depth of an augmented realitymessage such that the depth of the augmented reality message is equal todepth of the reference point. As a result, the HMD may display anaugmented reality message such that the augmented reality message islocated in the same plane as the reference point.

FIG. 5(b) is a view showing a method of displaying an augmented realitymessage using binocular disparity between a left image and a rightimage. The left subunit 10-1 of the HMD may display a left image, andthe right subunit 10-2 of the HMD may display a right image. The HMD maydisplay a left image corresponding to a first intersection point 53-1and a right image corresponding to a second intersection point 53-2.

The HMD may apply depth to an augmented reality message to be displayed.That is, the HMD may generate an augmented reality message to bedisplayed as a left image and a right image and apply binoculardisparity between the left image and the right image to convert theaugmented reality message into a three-dimensional image. As a result,the HMD may display an augmented reality message 55 having depth. Forexample, the HMD may provide an augmented reality message 55 located inthe same plane as the reference point to the user.

FIGS. 6(a) and 6(b) are views showing a method of correcting anintersection point of the HMD according to an embodiment of the presentspecification. Referring to FIG. 6(a), the HMD 10 may detect a referenceobject 61 and a hand 12 of a user. The HMD 10 may detect an event inwhich the hand 12 of the user touches a portion of the reference object61. The HMD 10 may decide a point of the reference object, on which theevent has been generated, as a reference point 62 using a camera unit 21and an image processing unit. The HMD 10 may decide a user's eye gazedirection in which the user gazes at the reference point 62 using an eyegaze detection unit 22. The HMD 10 may detect the position of a pupil 13of the user to decide the user's eye gaze direction.

The HMD 10 may set a first virtual line 63-1 linking the pupil 13 of theuser to the reference point 62 and calculate the position of a firstintersection point 64-1 between the first virtual line 63-1 and the HMD10. That is, the HMD 10 may perform eye gaze calibration based on theposition of the pupil 13 of the user and the position of the referencepoint 62 and calculate the position of the first intersection point 64-1as mentioned above as the result of the eye gaze calibration. The HMD 10may decide the position of a virtual object to be displayed on the HMDbased on the calculated position of the first intersection point 64-1.

A relative position between the HMD 10 and the pupil of the user may bechanged as the result of movement of the user. In this case, the HMD 10may detect that it is necessary to perform or re-perform eye gazecalibration. The HMD 10 may detect the change of the relative positionbetween the HMD 10 and the pupil of the user to correct the position ofan intersection point through eye gaze calibration. The HMD 10 maydetect the change of the relative position between the HMD 10 and thepupil of the user using a camera unit 22. In this case, the HMD 10 maycorrect the position of an intersection point through eye gazecalibration. Also, the HMD 10 may change the position of a virtualobject to be displayed based on the corrected portion of theintersection point.

Referring to FIG. 6(b), the relative position between the HMD 10 and thepupil of the user is lower than the relative position between the HMD 10and the pupil of the user shown in FIG. 6(a). In a case in which therelative position between the HMD 10 and the pupil of the user ischanged, the HMD 10 may set a second virtual line 63-2 linking the pupil13 of the user to the reference point 62 and calculate the position of asecond intersection point 64-2 between the first virtual line 63-2 andthe HMD 10. That is, the HMD 10 may perform eye gaze calibration basedon the position of the pupil 13 of the user and the position of thereference point 62 and calculate the position of the second intersectionpoint 64-2 as mentioned above as the result of the eye gaze calibration.The HMD 10 may decide the position of a virtual object to be displayedon the HMD 10 based on the calculated position of the secondintersection point 64-2. Through the above process, the HMD 10 maycorrect the position of the intersection point from the firstintersection point 64-1 to the second intersection point 64-2 to decidethe position of a virtual object to be displayed such that the positionof the virtual object corresponds to the user's eye gaze. Consequently,the embodiment of the present specification has the effect of adaptivelydeciding the position of a virtual object based on the relative positionbetween the HMD and the pupil of the user.

FIG. 7 is a view showing an eye gaze calibration method of an HMD usinga moving object according to an embodiment of the present specification.The HMD may detect a reference object as reference of eye gazecalibration and a user's eye gaze direction to perform eye gazecalibration. The HMD may detect moving objects having mobility in frontof the HMD using a camera unit. The HMD may compare routes of the movingobjects with a route along which user's eye gaze moves to decide areference object and then perform eye gaze calibration. According toembodiments, the HMD may decide a reference point, which is a portion ofthe reference object, as reference of eye gaze calibration. Hereinafter,the reference object will be described. However, such a description maybe similarly applied to the reference point, which is a portion of thereference object.

The HMD 10 may detect at least one moving object in front of the HMD.The moving object is an object having mobility, the position of which ischanged over time. The moving object may be a real object present in areal world, not a virtual object displayed on the display device. Asshown in FIG. 7, the HMD may detect a first moving object 71 and asecond moving object 72 through a camera unit 73. The first movingobject 71 may move along a first route and the second moving object 72may move along a second route. The HMD may detect the first route, alongwhich the first moving object 71 moves, and the second route, alongwhich the second moving object 72 moves, through the camera unit 73. Asshown at the upper end of FIG. 7, the first moving object 71 and thesecond moving object 72 may move in opposite directions. Consequently,the first route of the first moving object 71 and the second route ofthe second moving object 72 may extend in opposite direction.

When a plurality of moving objects is detected through the camera unit73, the HMD may include only moving objects satisfying at least oneselected from among a velocity condition, a size condition, a movingdirection condition, and a color condition in a reference objectcandidate group. The velocity condition may include a velocity thresholdrange. In a case in which the velocity of a moving object is too fast ortoo slow, it may be difficult for the HMD to decide a route of themoving object. Consequently, the HMD may include only moving objectshaving velocities within the velocity threshold range in the referenceobject candidate group. In addition, the size condition may include asize threshold range. Consequently, the HMD may include only movingobjects having sizes within the size threshold range in the referenceobject candidate group. In addition, the moving direction condition mayinclude a direction threshold range. In a case in which a moving objectmoves toward the HMD or moves far from the HMD, it may be difficult forthe HMD to correctly decide a route of the moving object. Consequently,the HMD may include only moving objects moving in directions within thedirection threshold range in the reference object candidate group. Inaddition, the color condition may be a condition in which eye gazecalibration is performed for specific colors. Consequently, the HMD mayinclude only moving objects having colors included in the colorcondition in the reference object candidate group. For example, the HMDmay include only red moving objects attracting user attention in thereference object candidate group. In this case, the user may recognizewhat colored objects are to be gazed at for eye gaze calibration andthen perform the eye gaze calibration.

In FIG. 7, in a case in which the first moving object and the secondmoving object satisfy at least one selected from among the velocitycondition, the size condition, the moving direction condition, and thecolor condition, the HMD may include the first moving object and thesecond moving object in the reference object candidate group and decidea reference object from among the moving objects included in thereference object candidate group.

The HMD may detect a user's eye gaze direction through an eye gazedetection unit 74. The HMD may detect a third route along which user'seye gaze moves. At the upper end of FIG. 7, the HMD may detect a firstintersection point 75-1 at which the HMD intersects the user's eye gazedirection using the eye gaze detection unit 74. In addition, at thelower end of FIG. 7, the HMD may detect a second intersection point 75-2at which the HMD intersects the user's eye gaze direction using the eyegaze detection unit 74. The HMD may decide the third route, along whichthe user's eye gaze moves, based on the fact that the first intersectionpoint 75-1 has moved to second intersection point 75-2.

The HMD may compare the detected first route and the detected secondroute with the third route, along which the user's eye gaze moves, usingan image processing unit. In a case in which a moving object having aroute generally coinciding with the third route is present, the HMD maydetermine that the user gazes at the corresponding moving object.Consequently, the HMD may set the moving object having the routegenerally coinciding with the third route, along which the user's eyegaze moves, as a reference object. In FIG. 7, the HMD may determine thatthe first route of the first moving object is substantially identicalwith the third route of the user's eye gaze. Coincidence between the tworoutes may include a case in which the two routes are substantiallyidentical with each other in terms of a direction and a moving rate inaddition to a case in which the two routes are completely identical witheach other. Consequently, the HMD may decide the first moving object,moving along the first route coinciding with the third route, as areference object.

The HMD 10 may perform eye gaze calibration based on the position of thedecided reference object and the position of a pupil of the user.Alternatively, the HMD 10 may perform eye gaze calibration based on theposition of the decided reference point and the position of the pupil ofthe user. In a case in which the HMD 10 performs eye gaze calibrationbased on the position of the reference point, the eye gaze calibrationmay be performed with higher accuracy than in a case in which the HMD 10performs eye gaze calibration based on the position of the referenceobject.

The HMD 10 may set a virtual line 76 linking the pupil 13 of the user tothe reference object 71 and calculate the intersection point 75-1 or75-2 between the virtual line 76 and the HMD. The HMD 10 may perform eyegaze calibration based on the calculated position of the intersectionpoint. In a case in which the reference object moves from a firstposition to a second position, the HMD may detect a user's first eyegaze direction when the reference object is at the first position,detect a user's second eye gaze direction when the reference object isat the second position, and perform eye gaze calibration based on thefirst position of the reference object, the second position of thereference object, the first eye gaze direction, and the second eye gazedirection. In another embodiment, in a case in which the referenceobject moves from a first position to a second position, the HMD maycontinuously perform eye gaze calibration based on the position of thereference object and user's eye gaze moving along the reference object.

The HMD may decide the position of digital information to be displayedon the HMD according to a result of the eye gaze calibration. Inaddition, the HMD 10 may move the position of the digital informationdisplayed on the HMD according to the result of the eye gaze calibrationand then display the digital information.

In this way, the HMD 10 may perform eye gaze calibration using a routeof user's eye gaze and a route of a moving object having mobility amongreal objects in a real world. Consequently, the HMD may perform eye gazecalibration while the user wears and uses the HMD 10 without using anadditional calibration screen or calibration time.

FIG. 8 is a view showing an eye gaze calibration method of an HMD usinga moving object according to another embodiment of the presentspecification. The HMD may detect a reference object as reference of eyegaze calibration and a user's eye gaze direction to perform eye gazecalibration. The HMD may detect moving objects having mobility in frontof the HMD using a camera unit. The HMD may compare routes of the movingobjects with a route along which user's eye gaze moves to decide areference object. The HMD may decide a reference point, which is aportion of the reference object, as reference of eye gaze calibration.The HMD may decide a portion of the reference object as a referencepoint according to point priority information. For example, in a case inwhich a reference object is an animal, the face of the animal may bedecided as a reference point. Particularly, in a case in which areference object is a human being, the face of the human being may bedecided as a reference point. On the other hand, in a case in which areference object is a car, the number plate, emblem, or headlight of thecar may be decided as a reference point. In addition, in a case in whicha place where eye gaze calibration is performed is dark, a brightportion of a reference object may be decided as a reference point.

The HMD 10 may detect at least one moving object in front of the HMD.The moving object is an object having mobility, the position of which ischanged over time. The moving object may be a real object present in areal world, not a virtual object displayed on the display device. Asshown at the upper end of FIG. 8, the HMD may detect a first movingobject 81 through a camera unit 83. In addition, the HMD may detect afirst moving point 82, which is a portion of the first moving object.The first moving point 82 may move along a first route. The HMD maydetect the first route, along which the first moving point 82 moves,through the camera unit 83.

The HMD may detect a user's eye gaze direction through an eye gazedetection unit 84. The HMD may detect a second route along which user'seye gaze moves. As previously described with reference to FIG. 7, theHMD may detect movement of an intersection point at which the HMDintersects the user's eye gaze direction to decide the second route.

The HMD may compare the detected first route of the first moving point82 with the second route, along which the user's eye gaze moves. In acase in which the first route generally is substantially identical withthe second route, the HMD may determine that the user gazes at the firstmoving point 82. Consequently, the HMD may set the first moving point 82having the route coinciding with the second route, along which theuser's eye gaze moves, as a reference point. Coincidence between the tworoutes may include a case in which the two routes generally coincidewith each other in terms of a direction and a moving rate in addition toa case in which the two routes completely coincide with each other.Consequently, the HMD may decide the first moving point 82, moving alongthe first route coinciding with the second route, as a reference point.

The HMD 10 may perform eye gaze calibration based on the position of thedecided reference point and the position of a pupil of the user. In acase in which the HMD 10 performs eye gaze calibration based on theposition of the reference point, the eye gaze calibration may beperformed with higher accuracy than in a case in which the HMD 10performs eye gaze calibration based on the position of the referenceobject.

The HMD 10 may set a virtual line 86 linking the pupil 13 of the user tothe first moving point 82 as the reference point and calculate anintersection point 85 between the virtual line 86 and the HMD. The HMD10 may perform eye gaze calibration based on the calculated position ofthe intersection point. The HMD may decide the position of digitalinformation to be displayed on the HMD according to a result of the eyegaze calibration. In addition, the HMD 10 may move the position of thedigital information displayed on the HMD according to the result of theeye gaze calibration and then display the digital information.

In this way, the HMD 10 may perform eye gaze calibration using a routeof user's eye gaze and a route of a moving object having mobility amongreal objects in a real world. Consequently, the HMD may perform eye gazecalibration while the user wears and uses the HMD 10 without using anadditional calibration screen or calibration time.

The lower end of FIG. 8 shows a method of deciding a reference object ina case in which a plurality of moving objects moves along the sameroute. The HMD 10 may detect a plurality of moving objects using thecamera unit 83. In a case in which a plurality of moving objects havingthe same route is detected by the camera unit, the HMD may set areference object from among the detected moving objects according toobject priority information. The object priority information may includeinformation regarding taste and tendency of a user. For example, in acase in which the user has a great interest in a car, when the HMDdetects a car and a truck moving along the same route, the HMD maydecide the car as a reference object. In addition, in a case in which adog 87 and a human being 81 moving along the same route are detected asshown at the lower end of FIG. 8, the HMD may decide the human being asa reference object.

In this way, the HMD may check taste and tendency of a user to decide amoving object which the user is predicted to gaze at as a referenceobject in advance and then may perform eye gaze calibration. As aresult, the HMD may correctly and rapidly decide a reference object asreference of eye gaze calibration.

FIG. 9 is a view showing a method of performing eye gaze calibrationwith respect to both eyes of a user using a reference point. The HMD mayinclude a left subunit 10-1 to provide a graphical user interface to aleft eye of the user and a right subunit 10-2 to provide a graphicaluser interface to a right eye of the user. Each subunit of the HMD mayinclude at least one selected from between a camera unit and an eye gazedetection unit. Hereinafter, particulars common to the left subunit andthe right subunit will be described in terms of the HMD.

As previously described with reference to FIG. 8, the HMD 10-1 and 10-2may detect a reference object 91. The HMD may recognize that thereference object 91 is a human being and decide the face of the humanbeing as a reference point 92. The HMD 10-1 and 10-2 may decide a user'seye gaze direction in which the user gazes at the reference point 92using an eye gaze detection unit. The HMD 10-1 and 10-2 may detect theposition of a pupil of the user to decide the user's eye gaze direction.The HMD may perform eye gaze calibration based on the position of thereference object 91 and the user's eye gaze direction. In a case inwhich the reference object 91 moves from a first position to a secondposition, the HMD may perform eye gaze calibration for the referenceobject at the first position and then perform eye gaze calibration forthe reference object at the second position. In addition, the HMD maycontinuously perform eye gaze calibration while the reference objectmoves from the first position to the second position. In addition, theHMD may display a virtual object according to a result of the eye gazecalibration. When the reference object moves from the first position tothe second position, the displayed virtual object may be moved from thefirst position to the second position according to movement of thereference object and then displayed.

At the upper end of FIG. 9, the left subunit 10-1 of the HMD may set afirst virtual line linking a pupil 13-1 of the left eye of the user tothe reference point 92 of the reference object 91 at the first positionand calculate a first intersection point 93-1 between the first virtualline and the left subunit 10-1. That is, the HMD may perform eye gazecalibration based on the position of the pupil 13-1 of the left eye ofthe user and the position of the reference point 92 and calculate thefirst intersection point 93-1 as mentioned above as a result of the eyegaze calibration. The HMD may decide the position of a left image to bedisplayed on the left subunit 10-1 based on the calculated position ofthe first intersection point 93-1. That is, the HMD may move theposition of the left image according to the result of the eye gazecalibration and then display the left image.

The right subunit 10-2 of the HMD may set a second virtual line linkinga pupil 13-2 of the right eye of the user to the reference point 92 ofthe reference object 91 at the first position and calculate a secondintersection point 93-2 between the second virtual line and the rightsubunit 10-2. That is, the HMD may perform eye gaze calibration based onthe position of the pupil 13-2 of the right eye of the user and theposition of the reference point 92 and calculate the second intersectionpoint 93-2 as mentioned above as a result of the eye gaze calibration.The HMD may decide the position of a right image to be displayed on theright subunit 10-2 based on the calculated position of the secondintersection point 93-2. That is, the HMD may move the position of theright image according to the result of the eye gaze calibration and thendisplay the position of the right image.

The HMD may measure the distance from the HMD to the reference point 92using a distance measurement unit. The distance measurement unit maymeasure the distance from the HMD to the reference point using at leastone selected from among an ultrasonic sensor, a laser sensor, and aninfrared sensor. The MID may decide depth of an augmented realitymessage based on the measured distance from the HMD to the referencepoint. For example, the HMD may set depth of an augmented realitymessage such that the depth of the augmented reality message is equal todepth of the reference point. As a result, the HMD may display anaugmented reality message such that the augmented reality message islocated in the same plane as the reference point with the same depth asthe reference point.

The HMD may display an augmented reality message using binoculardisparity between a left image and a right image. The left subunit 10-1of the HMD may display a left image and the right subunit 10-2 of theHMD may display a right image. The HMD may display a left imagecorresponding to the first intersection point 93-1 and a right imagecorresponding to the second intersection point 93-2.

The HMD may apply depth to an augmented reality message to be displayed.That is, the HMD may generate an augmented reality message to bedisplayed as a left image and a right image and apply binoculardisparity between the left image and the right image to convert theaugmented reality message into a three-dimensional image. As a result,the HMD may display an augmented reality message 94 having depth. Forexample, in a case in which the depth of the reference point of thereference object located at the first position is a first depth, the HMDmay provide the augmented reality message 94 located at the first depthto the user.

At the lower end of FIG. 9, the left subunit 10-1 of the HMD may set afirst virtual line linking the pupil 13-1 of the left eye of the user tothe reference point 92 of the reference object 91 at the second positionand calculate a third intersection point 93-3 between the first virtualline and the left subunit 10-1. That is, the HMD may perform eye gazecalibration based on the position of the pupil 13-1 of the left eye ofthe user and the position of the reference point 92 and calculate thethird intersection point 93-3 as mentioned above as a result of the eyegaze calibration. The HMD may decide the position of a left image to bedisplayed on the left subunit 10-1 based on the calculated position ofthe third intersection point 93-3. That is, the HMD may move theposition of the left image according to the result of the eye gazecalibration and then display the left image.

The right subunit 10-2 of the HMD may set a second virtual line linkingthe pupil 13-2 of the right eye of the user to the reference point 92 ofthe reference object 91 at the second position and calculate a fourthintersection point 93-4 between the second virtual line and the rightsubunit 10-2. That is, the HMD may perform eye gaze calibration based onthe position of the pupil 13-2 of the right eye of the user and theposition of the reference point 92 and calculate the fourth intersectionpoint 93-4 as mentioned above as a result of the eye gaze calibration.The HMD may decide the position of a right image to be displayed on theright subunit 10-2 based on the calculated position of the fourthintersection point 93-4. That is, the HMD may move the position of theright image according to the result of the eye gaze calibration and thendisplay the right image.

The HMD may display an augmented reality message using binoculardisparity between a left image and a right image. The left subunit 10-1of the HMD may display a left image and the right subunit 10-2 of theHMD may display a right image. The HMD may display a left imagecorresponding to the third intersection point 93-3 and a right imagecorresponding to the fourth intersection point 93-4.

The HMD may apply depth to an augmented reality message to be displayed.That is, the HMD may generate an augmented reality message to bedisplayed as a left image and a right image and apply binoculardisparity between the left image and the right image to convert theaugmented reality message into a three-dimensional image. As a result,the HMD may display an augmented reality message 94 having depth. Forexample, in a case in which the depth of the reference point of thereference object having moved to the second position is a second depth,the HMD may move the augmented reality message 94 located at the firstdepth to the second depth and then provide the augmented reality message94 located at the second depth to the user.

FIG. 10 is a block diagram showing an HMD according to an embodiment ofthe present specification. The HMD may include a camera unit 101, animage processing unit 102, an eye gaze detection unit 103, a displayunit 104, and a controller 105. In addition, the HMD may further includea distance measurement unit.

The camera unit 101 may sense an image of the surroundings located infront of the HMD. The camera unit 101 may include an image sensor. Theimage sensor may convert an optical signal into an electrical signal.The camera unit 101 may convert the sensed image of the surroundingsinto an electrical signal and transmit the electrical signal to thecontroller. The camera unit 101 may sense an image having a rangecorresponding to a visual field of a user. Consequently, the HMD maydecide which of the real objects contained in the surroundings the usergazes at based on the sensed image and the position of a pupil of theuser.

The image processing unit 102 may distinguish a real object from theother object contained in the sensed image of the surroundings. Theimage processing unit 102 may detect at least one real object containedin the image of the surroundings as an object unit. Also, the imageprocessing unit 102 may detect a hand of the user from the sensed imageof the surroundings. The image processing unit 102 may distinguish thehand of the user from a hand of another person using at least one of ashape and a gesture of the hand of the user. The image processing unit102 may detect an event of the hand of the user on a real object anddecide the real object, on which the event has been detected, as areference object. Also, the image processing unit 102 may decide aportion of the reference object, on which the event has been detected,as a reference point. The image processing unit 102 may transmitinformation regarding at least one of the decided reference object andthe decided reference point to the controller.

The eye gaze detection unit 103 may detect a user's eye gaze direction.The eye gaze detection unit 103 may detect a user's eye gaze directionbased on movement of a pupil of the user. Also, the eye gaze detectionunit 103 may detect whether eye gaze calibration is to be re-performedbased on the relative position between the HMD and the pupil of theuser. The eye gaze detection unit 103 may provide information regardingthe user's eye gaze direction to the controller.

The display unit 104 may display a virtual object. The virtual objectmay include a graphical user interface provided to the user by thedisplay unit. The display unit 104 may display a left image and a rightimage on a left eye and a right eye of the user, respectively. Thedisplay unit 104 may display a virtual object having depth usingbinocular disparity. The display unit 104 may control the position of avirtual object to be displayed and display the virtual object accordingto the result of the eye gaze calibration. The display unit 104 mayinclude a transparent optical display unit. Consequently, the user mayrecognize the surroundings from a visible light transmitted through thedisplay unit 104.

The distance measurement unit may measure the distance from the HMD to areference object. Also, the distance measurement unit may measure thedistance from the HMD to a reference point. The HMD may decide depth ofa virtual object to be displayed based on the distance from the HMD tothe reference object or the reference point measured by the distancemeasurement unit. The distance measurement unit may measure the distancefrom the HMD to the reference object or the reference point using atleast one selected from among an ultrasonic sensor, a laser sensor, andan infrared sensor. The distance measurement unit may transmitinformation regarding the measured distance from the HMD to thereference object or the reference point to the controller. Inembodiments of the present specification, the distance measurement unitmay be an optional element of the HMD.

The controller 105 may perform an application and process data in theHMD. The controller 105 may control the camera unit 101, the imageprocessing unit 102, the eye gaze detection unit 103, and the displayunit 104. Also, the controller 105 may control transmission andreception of data between the above units. In embodiments of the presentspecification, the controller 105 may receive an image of thesurroundings from the camera unit 101. Also, the controller 105 mayreceive information regarding a reference object, a reference point,event generation, and position of a hand of a user from the imageprocessing unit 102. Also, the controller 105 may receive informationregarding a user's eye gaze direction from the eye gaze detection unit103. The controller 105 may perform eye gaze calibration using theinformation regarding the reference object and the user's eye gazedirection. Also, the controller 105 may perform eye gaze calibrationusing the information regarding the reference point and the user's eyegaze direction. The controller 105 may control the position of a virtualobject displayed on the display unit 104 according to the result of theeye gaze calibration.

FIG. 10 is a block diagram of an embodiment of the presentspecification. The respective blocks indicate elements of the HMD whichlogically distinguish therebetween. Consequently, the elements of theHMD may be incorporated into a single chip or a plurality of chipsaccording to design of the HMD.

FIG. 11 is a flowchart showing an eye gaze calibration method accordingto an embodiment of the present specification. The HMD may sense animage of the surroundings located in front of the HMD using the cameraunit (S10). The HMD may sense a range corresponding to a visual field ofa user.

The HMD may detect a reference object from the sensed image of thesurroundings using the image processing unit (S20). As previouslydescribed with reference to FIGS. 1 to 4, the HMD may distinguish atleast one real object contained in the sensed image of the surroundingsas an object unit. The HMD may detect a reference object from the atleast one real object distinguished as the object unit. A method of theHMD detecting a reference object will hereinafter be described in detailwith reference to FIGS. 9 and 10. Also, the HMD may detect a portion ofthe reference object as a reference point. In this case, the HMD mayperform eye gaze calibration using the position of the reference pointinstead of the position of the reference object such that the eye gazecalibration is accurately performed.

The HMD may decide a user's eye gaze direction in which a user gazes atthe detected reference object using the eye gaze detection unit (S30).The HMD may detect the position of a pupil of a user wearing the HMD todecide the user's eye gaze direction. The HMD may correct the user's eyegaze direction based on at least one of the relative position betweenthe HMD and the pupil of the user and the distance between the HMD andthe pupil of the user.

The HMD may perform eye gaze calibration based on the detected referenceobject and the detected user's eye gaze direction using the controller(S40). In a case in which the eye gaze calibration is performed usingthe reference object as previously described with reference to FIGS. 1to 4, the HMD may set a virtual line linking the pupil of the user tothe reference object using the position of the detected reference objectand the detected user's eye gaze direction. The HMD may calculate theposition of an intersection point between the virtual line and the HMD.That is, the HMD may perform eye gaze calibration based on the positionof the pupil of the user and the position of the reference object andcalculate the position of the intersection point as mentioned above asthe result of the eye gaze calibration. The HMD may decide the positionof a virtual object to be displayed on the HMD based on the calculatedposition of the intersection point.

In a case in which the eye gaze calibration is performed using thereference point as another embodiment, the HMD may set a virtual linelinking the pupil of the user to the reference point using the positionof the detected reference point and the detected user's eye gazedirection. The HMD may calculate the position of an intersection pointbetween the virtual line and the HMD. That is, the HMD may perform eyegaze calibration based on the position of the pupil of the user and theposition of the reference point and calculate the position of theintersection point as mentioned above as the result of the eye gazecalibration. The HMD may decide the position of a virtual object to bedisplayed on the HMD based on the calculated position of theintersection point.

The virtual object may include an object displayed on the display unitof the HMD. For example, the HMD may display virtual objects, such as anapplication icon, an alarm message, a navigation direction indicator,and a notification message, as well as an augmented reality (AR) messagedescribed with reference to FIG. 5 on the display unit.

According to embodiments, the HMD may perform eye gaze calibration onlyin a case in which the difference between the direction of the referenceobject and the user's eye gaze direction in which the user gazes at thereference object is within an allowable error range. In a case in whichthe difference between the direction of the reference object and theuser's eye gaze direction in which the user gazes at the referenceobject is out of the allowable error range, on the other hand, the HMDmay suspend eye gaze calibration in order to prevent the occurrence ofan error of eye gaze calibration.

FIG. 12 is a flowchart showing a method of deciding a reference objectaccording to an embodiment of the present specification. The HMD maydetect at least one object contained in the sensed image of thesurroundings as a first object (S110). As previously described withreference to FIGS. 1 to 4, the HMD may extract at least one real objectfrom the image of the surroundings using the image processing unit. TheHMD may detect the extracted at least one real object as a first object.Alternatively, the HMD may detect two or more real objects as firstobjects.

The HMD may detect a hand of a user contained in the sensed image of thesurroundings as a second object (S120). As previously described withreference to FIGS. 1 to 4, the HMD may detect the hand of the user fromthe image of the surroundings using the image processing unit. Also, theHMD may detect the detected hand of the user as a second object. The HMDmay distinguish between the hand of the user and a hand of anotherperson using at least one of a shape of the hand of the user and agesture of the hand of the user. For example, when the HMD has detecteda V-shaped hand gesture of the user, the HMD may detect the hand of theuser as a second object. In another embodiment, the HMD may distinguishbetween the hand of the user and a hand of another person based on thelines of the palm of the user or the shape of a vein on the back of thehand of the user.

The HMD may detect an event of the second object with respect to thefirst object (S130). As previously described with reference to FIGS. 1to 4, the HMD may detect an operation of controlling the real objectdetected as the first object using the hand of the user detected as thesecond object through the image processing unit. The HMD may detect theabove operation as an event of the second object with respect to thefirst object.

For example, the HMD may detect an external device as a first object andthe tip of a finger of the user as a second object. In a case in whichthe first object is a keyboard input device, the HMD may detect anoperation of pressing a key of the keyboard as an event. Also, in a casein which the first object is a pen, the HMD may detect a writingoperation using a pen as an event. Also, in a case in which the firstobject is a device including a touch sensitive display, the HMD maydetect an operation of touching the touch sensitive display as an event.Also, in a case in which the first object is an electronic instrument,the HMD may detect an operation of controlling a control button of theelectronic instrument as an event. The control button of the electronicinstrument may include at least one selected from among a power button,volume control buttons, and channel control buttons.

The HMD may decide the first object, on which the event has beendetected, as a reference object (S140). As previously described withreference to FIGS. 1 to 4, the HMD may decide the first object, on whichthe event has been detected, as a reference object using the imageprocessing unit. That is, the HMD may detect one of the detected firstobjects, i.e. the first object, on which the event has been detected, asa reference object.

For example, in a case in which the user presses a key of a keyboard,the HMD may detect the keyboard as a reference object. Also, in a casein which the user performs a writing operation using a pen, the HMD maydetect the pen as a reference object. Also, in a case in which the usertouches a touch sensitive display, the HMD may detect a device includingthe touch sensitive display as a reference object. Also, in a case inwhich the user controls a control button of an electronic instrument,the HMD may detect the electronic instrument as a reference object.

Also, the HMD may detect a point of the reference object, on which theevent has been generated, as a reference point. In this case, the HMDmay perform eye gaze calibration using the position of the referencepoint instead of the position of the reference object such that the eyegaze calibration is accurately performed.

For example, in a case in which the reference object is a keyboard inputdevice, the HMD may detect a key of the keyboard pressed by the user asa reference point. Also, in a case in which the reference object is apen, the HMD may detect the tip of the pen gripped by the user as areference point. Also, in a case in which the reference object is adevice including a touch sensitive display, the HMD may detect a softbutton of the touch sensitive display, which the user touches, as areference object. Also, in a case in which the reference object is anelectronic instrument, the HMD may detect a control button of theelectronic instrument controlled by the user as a reference object.

The HMD may perform eye gaze calibration based on at least one of thedetected reference object and the detected reference point.

FIG. 13 is a flowchart showing a method of deciding a reference objectaccording to another embodiment of the present specification. The HMDmay detect a hand of a user contained in a sensed image of thesurroundings as a first object (S210). The HMD may detect the hand ofthe user before detecting a real object. As previously described withreference to FIGS. 1 to 4, the HMD may detect the hand of the user fromthe image of the surroundings using the image processing unit. Also, theHMD may detect the detected hand of the user as a first object. The HMDmay distinguish between the hand of the user and a hand of anotherperson using at least one of a shape of the hand of the user and agesture of the hand of the user. For example, when the HMD has detecteda V-shaped hand gesture of the user, the HMD may detect the hand of theuser as a first object. In another embodiment, the HMD may distinguishbetween the hand of the user and a hand of another person based on thelines of the palm of the user or the shape of a vein on the back of thehand of the user.

The HMD may detect at least one object contained in the sensed image ofthe surroundings as a second object (S220). As previously described withreference to FIGS. 1 to 4, the HMD may extract at least one real objectfrom the image of the surroundings using the image processing unit. TheHMD may detect the extracted at least one real object as a secondobject. Alternatively, the HMD may detect two or more real objects assecond objects.

The HMD may detect an event of the first object with respect to thesecond object (S230). As previously described with reference to FIGS. 1to 4, the HMD may detect an operation of controlling the real objectdetected as the second object using the hand of the user detected as thefirst object through the image processing unit. The HMD may detect theabove operation as an event of the first object with respect to thesecond object.

For example, the HMD may detect the tip of a finger of the user as afirst object and an external device as a second object. In a case inwhich the second object is a keyboard input device, the HMD may detectan operation of pressing a key of the keyboard as an event. Also, in acase in which the second object is a pen, the HMD may detect a writingoperation using a pen as an event. Also, in a case in which the secondobject is a device including a touch sensitive display, the HMD maydetect an operation of touching the touch sensitive display as an event.Also, in a case in which the second object is an electronic instrument,the HMD may detect an operation of controlling a control button of theelectronic instrument as an event. The control button of the electronicinstrument may include at least one selected from among a power button,volume control buttons, and channel control buttons.

The HMD may decide the second object, on which the event has beendetected, as a reference object (S240). As previously described withreference to FIGS. 1 to 4, the HMD may decide the second object, onwhich the event has been detected, as a reference object using the imageprocessing unit. That is, the HMD may detect one of the detected secondobjects, i.e. the second object, on which the event has been detected,as a reference object.

For example, in a case in which the user presses a key of a keyboard,the HMD may detect the keyboard as a reference object. Also, in a casein which the user performs a writing operation using a pen, the HMD maydetect the pen as a reference object. Also, in a case in which the usertouches a touch sensitive display, the HMD may detect a device includingthe touch sensitive display as a reference object. Also, in a case inwhich the user controls a control button of an electronic instrument,the HMD may detect the electronic instrument as a reference object.

Also, the HMD may detect a point of the reference object, on which theevent has been generated, as a reference point. In this case, the HMDmay perform eye gaze calibration using the position of the referencepoint instead of the position of the reference object such that the eyegaze calibration is accurately performed.

For example, in a case in which the reference object is a keyboard inputdevice, the HMD may detect a key of the keyboard pressed by the user asa reference point. Also, in a case in which the reference object is apen, the HMD may detect the tip of the pen gripped by the user as areference point. Also, in a case in which the reference object is adevice including a touch sensitive display, the HMD may detect a softbutton of the touch sensitive display, which the user touches, as areference object. Also, in a case in which the reference object is anelectronic instrument, the HMD may detect a control button of theelectronic instrument controlled by the user as a reference object.

The HMD may perform eye gaze calibration based on at least one of thedetected reference object and the detected reference point. The HMD maymove the position of an image or a virtual object to be displayedaccording to a result of the eye gaze calibration.

FIG. 14 is a flowchart showing a method of deciding a reference objectaccording to a further embodiment of the present specification. The HMDmay detect a first route of a first moving object and a second route ofa second moving object in front of the HMD (S310). As previouslydescribed with reference to FIG. 7, the HMD may detect a route of amoving object having mobility. When a plurality of moving objects isdetected through a camera unit, the HMD may include only moving objectssatisfying at least one selected from among a velocity condition, a sizecondition, a moving direction condition, and a color condition in areference object candidate group. The camera unit may detect an imagehaving the same range as a visual field of the user. In a case in whichthe first moving object and the second moving object satisfy at leastone of the above conditions, the HMD may include the first moving objectand the second moving object in the reference object candidate group. Inaddition, the HMD may detect a first route of the first moving objectincluding a moving direction and displacement of the first moving objectand a second route of the second moving object including a movingdirection and displacement of the second moving object.

The HMD may detect a third route along which user's eye gaze moves(S320). The HMD may detect a user's eye gaze direction based on movementof a pupil of the user. The HMD may track user's eye gaze moving overtime to detect the third route, along which the user's eye gaze moves.The HMD may detect the third route of the user's eye gaze using an eyegaze detection unit. The HMD may compare the detected first route andthe detected second route with the third route.

In a case in which the detected first route is substantially identicalwith the detected third route, the HMD may set the first moving objectas a reference object (S330). The HMD may compare the direction,velocity, and movement distance of the first moving object moving alongthe first route with the direction, velocity, and movement distance ofthe user's eye gaze to determine whether the first route issubstantially identical with the third route. Even in a case in whichthe first route of the reference object and the third route of theuser's eye gaze are substantially identical with each other by a firstrate or more in addition to a case in which the first route and thethird route completely coincide with each other, the HMD may determinethat the first route is substantially identical with the third route.Only when the first route and the third route coincide with each otherby the first rate or more, the HMD may perform the following eye gazecalibration.

In a case in which the detected second route is substantially identicalwith the detected third route, the HMD may set the second moving objectas a reference object (S340). The HMD may compare the direction,velocity, and movement distance of the second moving object moving alongthe second route with the direction, velocity, and movement distance ofthe user's eye gaze to determine whether the second route issubstantially identical with the third route. Even in a case in whichthe second route of the reference object and the third route of theuser's eye gaze coincide with each other by the first rate or more inaddition to a case in which the second route and the third routecompletely coincide with each other, the HMD may determine that thesecond route is substantially identical with the third route. Only whenthe second route and the third route coincide with each other by thefirst rate or more, the HMD may perform the following eye gazecalibration.

The HMD may perform eye gaze calibration based on the route of the setreference object and the detected third route (S350). In a case in whichthe wearing position of the HMD is changed or a case in which the changeof user's eye gaze exceeds a threshold level, the HMD may perform eyegaze calibration. The HMD may move the position of a virtual object orvisual information which has been displayed according to a result of theeye gaze calibration. In addition, the HMD may display a virtual objector visual information which has not yet been displayed at a positionacquired by reflecting the result of the eye gaze calibration.

According to embodiments of the present specification, it is possiblefor an HMD to perform eye gaze calibration while a user uses the HMD.

Also, according to embodiments of the present specification, it ispossible for an HMD to use a real object contained in the surroundingsas a reference point for eye gaze calibration.

According to embodiments of the present specification, it is possiblefor an HMD to recognize a portion of the body of a user and to use therecognized portion of the body of the user as a reference point for eyegaze calibration.

According to embodiments of the present specification, it is possiblefor an HMD to use a hand of a user as a reference point for eye gazecalibration.

According to embodiments of the present specification, it is possiblefor an HMD to detect an event on a real object and to use the realobject as a reference point for eye gaze calibration.

According to embodiments of the present specification, it is possiblefor an HMD to detect a user's eye gaze direction and to use the detecteduser's eye gaze direction in eye gaze calibration.

According to embodiments of the present specification, it is possiblefor an HMD to measure the distance from the HMD to a reference point andto decide depth of a virtual object displayed on the HMD based on themeasured distance from the HMD to the reference point.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the present specificationwithout departing from the spirit or scope of the specifications. Thus,it is intended that the present specification covers the modificationsand variations of this specification provided they come within the scopeof the appended claims and their equivalents.

What is claimed is:
 1. A head mounted display (HMD) comprising: a cameraunit configured to detect a first route of a first moving object and asecond route of a second moving object in front of the HMD; a displayunit configured to display visual information; an eye gaze detectionunit configured to detect a third route along which an eye gaze of auser moves; and a controller configured to perform an eye gazecalibration based on a route of a reference object and the detectedthird route, wherein the controller is further configured to: select thereference object from a plurality of real moving objects including thefirst moving object and the second moving object, wherein the referenceobject is selected by comparing the third route of the eye gaze witheach route of the plurality of real moving objects, wherein theplurality of real moving objects are not images displayed by the displayunit of the head mounted display; change a position of the displayedvisual image according to the result of the eye gaze calibration; if thedetected first route is substantially identical with the detected thirdroute, set the first moving object as the reference object; and if thedetected second route is substantially identical with the detected thirdroute, set the second moving object as the reference object.
 2. The HMDaccording to claim 1, further comprising: a display unit configured todisplay visual information, wherein the controller further changes aposition of the displayed visual information according to a result ofthe eye gaze calibration.
 3. The HMD according to claim 2, wherein thevisual information is displayed between an eye of the user and thereference object.
 4. The HMD according to claim 1, wherein, if aplurality of moving objects having the same route is detected by thecamera unit, the controller sets the reference object from among thedetected moving objects according to object priority information.
 5. TheHMD according to claim 1, wherein the route of the reference object is aroute of a reference point located on the detected reference object, andthe reference point is a portion of the reference object.
 6. The HMDaccording to claim 5, wherein the reference point is decided as aportion of the reference object according to point priority information,and if the reference object is a human being, a face of the human beingis decided as the reference point.
 7. The HMD according to claim 1,wherein, if the reference object moves from a first position to a secondposition, the controller detects a user's first eye gaze direction whenthe reference object is at the first position, detects a user's secondeye gaze direction when the reference object is at the second position,and performs the eye gaze calibration based on the first position, thesecond position, the first eye gaze direction, and the second eye gazedirection.
 8. The HMD according to claim 1, wherein, if a position ofthe reference object moves from a first position to a second position,the controller continuously performs the eye gaze calibration based onthe position of the reference object and the user's eye gaze movementalong the reference object.
 9. The HMD according to claim 1, wherein, ifa plurality of moving objects is detected by the camera unit, thecontroller puts at least one of the moving objects satisfying at leastone selected from among a velocity condition, a size condition, a movingdirection condition, and a color condition in a reference objectcandidate group.
 10. The HMD according to claim 1, further comprising: adisplay unit configured to display a virtual object, wherein thecontroller further controls a position of the displayed virtual objectaccording to a result of the eye gaze calibration, and the virtualobject is a graphical user interface provided to the user through thedisplay unit.
 11. The HMD according to claim 10, wherein the displayunit displays the virtual object based on a position of the referenceobject using augmented reality technology.
 12. The HMD according toclaim 10, further comprising: a distance measurement unit configured tomeasure a first distance from the HMD to the detected reference object,wherein the controller further decides a display depth of the virtualobject in response to the measured first distance.
 13. The HMD accordingto claim 12, wherein the display unit provides a left eye and a righteye of the user with a left image and a right image, respectively, andthe display unit expresses the depth of the virtual object usingbinocular disparity between the left image and the right image.
 14. TheHMD according to claim 12, wherein, if the reference object moves, thedisplay unit changes the depth of the virtual object in response to amoved position of the reference object.
 15. The HMD according to claim1, wherein the eye gaze calibration is performed when a wearing positionof the HMD is changed or when change of the user's eye gaze exceeds athreshold level.
 16. The HMD according to claim 1, wherein the eye gazedetection unit detects a user's eye gaze direction based on movement ofa pupil of the user.
 17. The HMD according to claim 1, wherein thecamera unit senses an image having the same range as a visual field ofthe user.
 18. The HMD according to claim 1, wherein the controllerperforms the eye gaze calibration only when the route of the referenceobject and the third route of the user's eye gaze are substantiallyidentical with each other by a first rate or more.
 19. The HMD accordingto claim 1, wherein the controller compares a direction, velocity, andmovement distance of the first moving object moving along the firstroute with a direction, velocity, and movement distance of the user'seye gaze to determine whether the detected first route is substantiallyidentical with the detected third route.
 20. A control method of a HMDcomprising: detecting, via a camera unit, a first route of a firstmoving object and a second route of a second moving object in front ofthe HMD; display visual information on a display unit; detecting, via aneye gaze detection unit, a third route along which user's eye gazemoves; selecting, via a controller, a reference object from a pluralityof real moving objects including the first moving object and the secondmoving object, wherein the reference object is selected by comparing thethird route of the eye gaze with each route of the plurality of realmoving objects, wherein the plurality of real moving objects are notimages displayed by the display unit of the head mounted display;changing, via the controller, a position of the displayed visualinformation according to a result of the eye gaze calibration; setting,via the controller, the first moving object as the reference object ifthe detected first route is substantially identical with the detectedthird route; setting, via the controller, the second moving object asthe reference object if the detected second route is substantiallyidentical with the detected third route; and performing, via thecontroller, an eye gaze calibration based on a route of the setreference object and the detected third route.